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ABSTRACT 

As more academic institutions spend larger sums of 
money to network their campuses and provide network connections to 
sources outside their campuses, questions emerge concerning the 
selection and configuration of appropriate network technologies and 
the appropriate types and levels of services to provide* 

Increasingly, academic administrators are asking questions about the 
benefits and impacts of networking* This paper reports on a study to 
develop and operationalize performance measures and indicators of the 
impacts of networking on the academic institution* The study examines 
information resources and services provided, organizational 
structures within which they are provided, the various classes of 
users, and users' activities on the network* Measurement and 
evaluation of networked information services is essential for 
administrators to justify such services and better meet user 
information needs* Approaches for evaluating networked information 
services are based on: (1) extensiveness — how much of the service has 
been provided; (2) efficiency — the use of resources in providing or 
accessing networked information services; (3) effectiveness — how well 
the networked information service met the objectives of the provider 
or user; and (4) impact — how a service made a difference in some 
other activity or situation* Performance measures and evaluation; key 
issues; attitudes, problems, and perceptions; possible performance 
measures; study progress; and the importance of the project are 
described* A brief discussion with the author is provided* (Contains 
16 references*) (Author/SWC) 
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Abstract 

A s more academic institutions spend larger sums 
of money to network their campuses and pro- 
vide network connections to sources outside their 
campuses, questions concerning the selection and 
configuration of appropriate network technologies, 
and the appropriate types and levels of services to 
provide, are emerging. Increasingly, academic 
administrators are asking questions about the bene- 
fits and impacts of networking. 

This paper reports on a study in progress 
designed to address these, and other questions relat- 
ed to assessing network technologies and services at 
academic institutions. The purpose of this study is 
to develop performance measures and indicators of 
the impacts of networking on the academic institu- 
tion. Key components of the academic networked 
environment will be defined and performance mea- 
sures of networked information technologies and 
services at academic institutions will be developed 
and operationalized. 

The paper provides an overview of the study’s 
progress to date, identifies and discusses key issues 
and preliminary findings that affect successful eval- 
uation of networked services and the development 
of performance measures, and describes an academ- 
ic networked environment performance measures 
manual that is currently under development. 

Introduction 

T he notion of the ‘academic networked environ- 
ment’ encompasses a range of campus electron- 
ic networked activities and services. Minimally, the 
academic networked environment includes informa- 
tion services, products, hardware and software, and 
resources which are received by campus users via 
electronic networks. In this environment, informa- 
tion services are provided by regional and national 
networks, although locally developed information 
services (ie. from the library, computing services, 
administration, individuals, or academic depart- 



ments) also comprise the academic networked envi- 
ronment. Both the networking of users and 
resources within the institution as well the cormect- 
ing of these users to other persons and resources 
outside of the institution are considered part of this 
environment. 

The notion of networked information services is 
an evolving one. Such services can be offered by 
individuals, libraries, computer centers, publishers, 
networks, government agencies, or a host of other 
organizations and groups with access to the Internet 
and the evolving National Information 
Infrastracture (Nil) (Huth and Gould, 1994) and the 
Global Information Infrastructure (GH) (Gore, 

1995). Networked information services comprise 
bulletin boards; email; list-servs; remote access to 
distant databases, software, and high speed comput- 
ing; and collaborative efforts among geographically 
dispersed individuals - to name but a few. A key 
aspect of ‘networked information services’ is that 
there are numerous providers - local and remote; 
there are a range of electronic information services 
available to users; and access to and use of these 
services continues to increase. 

Despite the fact that many institutions of higher 
education have built significant networks and are 
connected to the Internet and the evolving GII and 
Nil, there is little knowledge of how such coimec- 
tivity has affected the academic institution. Thus, 
some key questions are: 

- How much networking activity and of what 
types are taking place on a particular academic 
campus? 

- What types of users access the academic net 
work and to what types of services and activities 
do they cormect? 

- What are the costs for an academic network and 
various types of network activities and services? 

- How has access to and use of networked infor- 
mation resources and services affected teaching, 
research, learning, service, and other indicators 
of traditional academic performance? 
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To date, there has been little practical guidance 
offered to assess the impact of networidng on these 
traditional areas of academic institutional perfor- 
mance. Moreover, performance measures related to 
network use by specific audiences within the institu- 
tion such as faculty, administrators, librarians, stu- 
dents, and staff, are only now being developed. 

As nonprofit organizations implement new infor- 
mation technologies, they are beginning to caU for 
evaluation methods and measures to demonstrate 
that the resources invested in the new technologies 
have had some positive impact on their organiza- 
tions, the services they provide, and the users they 
serve. Traditional economic models that evaluate 
the impacts of information technology in terms of 
an organization’s bottom line are neither appropriate 
for nonprofit, service organizations, nor have they 
been used with much success in for-profit organiza- 
tions (Brynjolfsson, 1993; Computer Science and 
Telecommunications Board, 1994). 

There is evidence that a restructuring of the com- 
puting and communications infrastmcture as a result 
of the availability and use of electronic information 
is occurring and that this will have a fundamental 
impact on educational institutions. Already, this 
restmcturing is affecting the communication cus- 
toms and expectations of researchers in a variety of 
fields. In a larger sense, this restmcturing is affect- 
ing the entire information transfer cycle from the 
creation, stmcturing, and representation of informa- 
tion to its dissemination and use by the members of 
academic communities (McQure, 1993). 

A number of writers have attempted to describe 
the evolving academic networked environment and 
consider possible problems facing its development 
(Lynch, 1991; Drake, 1993). Recent reports issued 
by the Corporation for Public Broadcasting 
(1994) and the American Council on Education 
(1995) summarize current developments on uses 
and applications of information and networking 
technologies on campuses in the USA. But, to date, 
few formal efforts have been made to develop tech- 
niques to produce performance measures and assess 
the impact of networked information services on 
such an environment. 

Traditional criteria used in assessing information 
services may serve as a beginning model for net- 
worked information services assessment. For exam- 
ple, traditional performance indicators typically 
examine a service, activity, or product in terms of: 
extensiveness, effectiveness, efficiency, and impact 
(McClure, 1991). In addition, CAUSE, an associa- 
tion for managing and using information resources 
in higher education, published an excellent tool. 



Self-assessment for campus information technology 
services (Fleit, 1994) as well as Evaluation guide- 
lines for institutional information resources 
(CAUSE, 1995). Those in the process of assessing 
the academic networked environment may wish to 
review the self-assessment technique and the guide- 
lines developed by CAUSE. 

As described above, this is an exploratory study 
in progress and as such it is based on two broad 
research questions: 

- What is the academic networked environment? 

- What performance measures can be developed 
and tested to assess this academic networked 
environment? 

In answering the first question, the study examines 
information resources and services provided, orga- 
nizational structures within which they are provid- 
ed, the various classes of users involved, and users’ 
activities on the network. To answer the second 
question the study reviewed existing measures used 
at individual institutions and is developing and test- 
ing new measures of academic networking effec- 
tiveness, efficiency, and extensiveness, as well as 
impact. A core set of possible performance mea- 
sures as well as other evaluation techniques will be 
developed in a manual to assist those engaged in 
assessing academic networked environments. 

Because of the exploratory nature of this investi- 
gation, an inductive approach, using a variety of 
qualitative methods, is being taken. Among the 
methods being used are: focus groups, case studies, 
site visits, and interviews. Individuals involved in 
the design, implementation, support, and use of net- 
worked resources and services provide on-going 
feedback and comments on the study via an elec- 
tronic discussion list (see Appendix A). Individuals 
interested in learning more about the project and 
receiving updated project information are welcome 
to join the list. 

Performance Measures and 
Evaluation 

P erformance measures represent a broad manage- 
rial/evaluation concept that encompasses mea- 
surement of inputs (indicators of the resources 
essential to provide a service), outputs (indicators of 
the services resulting from the use of those 
resources), and impacts (the effect of these out- 
comes on other variables or factors). They are an 
essential means to assess the academic networked 
environment. Performance measures serve a number 
of useful purposes. They can: 
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- Identify those aspects of the network that are 
successful versus those aspects that are less suc- 
cessful 

- Provide trend-data to assess changes in the net- 
work and network services over time 

- Assist decision makers to allocate or reallocate 
resources and to plan for future network devel- 
opment 

- Monitor network activities and services to 
inform managers of any changes in activities or 
the quality of services 

- Determine the degree to which users are satisfied 
with the network and network services 

- Assist network managers to justify expenditures 
and be accountable for those expenditures 

Simply stated, performance measures ask decision- 
makers to answer the question: How well is the ser- 
vice or activity doing what it claims to be doing? 

Performance measures also assist managers to 
formally evaluate the network. Thus, evaluation is 
the process of identifying and collecting data about 
specific services or activities, establishing criteria to 
assess their success, and determining the degree to 
which the service or activity accomplishes stated 
objectives. As such, evaluation is a decision-making 
tool intended primarily to assist decision-makers 
allocate resources that best accomplish organiza- 
tional goals. Evaluation reflects value judgements 
on the part of the evaluator regarding the adequacy, 
appropriateness, and success of a particular service 
or activity. 

In a broader organizational context, measurement 
and evaluation of networked information services 
are essential for resource allocation, planning, and 
improving services. Without measures that can eval- 
uate particular services, decision-makers must rely 
on intuition and anecdotal information as a basis for 
assessing the usefulness and value of a particular 
service. Perhaps most importantly, measurement 
and evaluation provide feedback for users to make 
known how well those services meet their needs. 

Approaches for evaluating networked informa- 
tion services can be based on the following criteria: 

- Extensiveness : how much of the service has 
been provided, eg. number of users logging-in 
per week on a bulletin board, or the number of 
participants of a particular list-serv 

- Efficiency : the use of resources in providing or 
accessing networked information services, eg. 
cost per session in providing access to remote 
users of an online catalog, or average time 



required to successfully telnet to a remote data- 
base 

- Effectiveness : how well the networked informa- 
tion service met the objectives of the provider or 
of the user, eg. success rate of identifying and 
accessing the information needed by the user 

- Impact : how a service made a difference in 
some other activity or situation, eg. the degree to 
which faculty network users increased their 
research productivity or teaching effectiveness 
by use of networked information services 

Although evaluations of networked information ser- 
vices need to consider extensiveness and efficiency 
criteria, much more attention needs to be given to 
effectiveness and impact measures. As will be dis- 
cussed later in this paper, however, developing mea- 
sures of impacts ft’om networked services remains a 
very difficult task. 

Because networked information services are 
multi-dimensional, the type of evaluation needed 
typically will be multi-dimensional. A single mea- 
sure provides only one ‘snapshot’ of a particular 
service; multiple ‘snapshots’ from different mea- 
sures are needed. Moreover, evaluators of net- 
worked information services will need to know 
what type of evaluation approach and data collec- 
tion techniques will be appropriate for what types of 
services (McQure, 1994). An important point, 
however, is that researchers need to develop evalua- 
tion strategies that are user-based, that is, they 
examine networked information services from the 
point-of-view of the user. 

Providers of networked information services must 
not accept as a ‘given’ that their services, resources, 
and technical procedures are efficient and effective; 
rather, they must test their assumptions about the 
quality of networked information services through 
an ongoing process of evaluation. Ongoing evalua- 
tion activities are essential to support the provider’s 
planning process. Planning and evaluation are 
two sides of the same coin. Each will be more suc- 
cessful when the other is part of the overall services 
design and implementation approach. 

Developing, operationalizing, and validating a 
range of performance measures that encourages an 
academic institution to assess what types of net- 
worked information services have what level of 
quality, have what impacts on the educational 
process, and have what costs is essential if adminis- 
trators of networked information in the academic 
setting are to justify such services and better meet 
user information needs. 
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Selected Key Issues 

A t this writing, a number of site visits, focus 
groups, and small group interviews have been 
conducted at professional conferences and in select- 
ed academic institutions. Participants were academ- 
ic computing professionals, librarians, and others 
from a variety of institutions. The primary intent in 
conducting data collection activities was to inform 
the study team’s understanding of the research ques- 
tions, the state of modeling and evaluation of acad- 
emic netwoiicing in practice, and to obtain assess- 
ments of draft performance measures imder devel- 
opment by the study team. The key issues which 
emerged from data collection activities to date 
include the following. 

DRIVERS OF THE DEVELOPMENT OF NETWORK 
TECHNOLOGIES AND SERVICES 

Participants identified changes in users’ expecta- 
tions and experiences, changes in technology, and 
changes in the nature of educational processes, and 
institutional support for those processes, as drivers 
of the development of networking on their campus- 
es. 

Both faculty and students now seem to expect 
that network access should, and will, be available at 
all times, and from a variety of locations. Such 
expectations have been instrumental in causing net- 
work providers to accelerate their planning and 
implementation schedules. This has been particular- 
ly true in the case of incoming freshmen. One acad- 
emic administrator commented: 

‘Seeing a whole new crop of freshmen come in, 
computer literate in a way that we had never seen 
before ... aU of a sudden this class came in and 
said, ‘This is our god given right and why isn’t 
there a connection in every dorm room?” We had 
a plan to have that in a year and a half, and we . . . 
have just spent the last two months wildly coming 
up with a plan to make sure that we got it by 
September, 1995, because student demand is 
there.’ 

In addition, there is a recognition among university 
administrators that a network is essential in order to 
attract faculty. As one participant explained, quoting 
a university provost, ‘The world is now very differ- 
ent and every faculty person we recruit needs a 
dowry, needs to understand what kind of worksta- 
tion they are going to have on their desk, what kind 
of networking connection.’ 



BARRIERS TO THE DEVELOPMENT OF NETWORK 
TECHNOLOGIES AND SERVICES 

In spite of the increasing demand for networking 
and growing recognition on the part of administra- 
tors of the importance of networking, a number of 
barriers to the growth and development of network- 
ing exist. Among these are problems associated 
with network technologies and pedagogical limita- 
tions. 

A major challenge to network managers is ‘ . . . 
getting our systems to be easy to use. They’re still 
not good enough. They’re not like dialing a tele- 
phone to get what you need ... the systems are not 
intuitive and easy to use. And there are a trillion dif- 
ferent kinds of systems and almost a trillion differ- 
ent interfaces to access them ’ 

Another aspect of the network technology which 
stands in the way of development is its distributed 
nature and the problems that creates. ‘Anyone can 
get an Internet address and hang a server on the net- 
work. There are a whole lot of issues that the main- 
frame administrator used to handle. It’s now distrib- 
uted all over the network. And if they don’t manage 
it properly . . . your whole network is exposed.’ 

This distributed environment presents an image, 
oftentimes, that no one is really in charge or in con- 
trol of networking developments on campus - these 
developments just ‘happen’. 

Yet another barrier to the development of net- 
working is the lack of appropriate pedagogical mod- 
els to take advantage of the technology. ‘People are 
fundamentally automating old things . . . most of 
our professors haven’t really internalized how to use 
the technology to really change the way they con- 
duct their classes.’ The full advantage of network- 
ing may not be realized until new pedagogical mod- 
els are developed. By this, the authors mean that it 
may be too early to measure impacts of networking 
on the academic institution since traditional models 
of teaching and learning are stiU in use. New mod- 
els for teaching and learning that exploit the 
networked environment are stiU being developed. 

As one interviewee commented, ‘We are aU stiU 
floundering a bit as to how best to use and apply 
networking services.’ 

A final issue which was identified during the data 
coUection was that of measures of the impacts of 
networking. Participants discussed financial mea- 
sures and impact measures and agreed that, whUe 
traditional measures of technology impacts are often 
inappropriate, new measures are yet to be devel- 
oped. ‘AU ofthe traditional models, aU of the 
accounting models, just don’t apply any more . . . 
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[we need] to develop new models, and these models 
are going to be squishier.’ 

There was also some suggestion that academic 
administrators don’t ask for economic justification 
of requests for investment in new technologies, or in 
other improvements to the institution. ‘Colleges and 
universities don’t make these decisions based on 
direct economic issues. They don’t ask what is the 
cost benefit . . . they don’t ask that with respect to 
anything they do. I mean there is no bottom line, 
there isn’t.’ 

THE NEED FOR MEASURES AND ASSESSMENT 

Moving beyond financial measures, participants dis- 
cussed a developing interest in measures of the 
impacts of networking on teaching and learning. 
Administrators are beginning to ask questions like, 
‘Has it enabled an instructor to increase the contents 
or broaden the contents or get deeper in the content 
of the class?’ and ‘Has it reduced their administra- 
tive work in administering the class?’ 

In response to such questions, network adminis- 
trators are beginning to develop and apply new mea- 
sures. For example, ‘We provided multimedia sup- 
port for classrooms where we have evidence that the 
faculty member is spending less time on the 
mechanics and more on the content. So that the stu- 
dents are getting more content and learning it 
faster.’ However, most evidence of networking’s 
impacts on teaching and learning is anecdotal. 

For example, one faculty member commented, 
‘We’ve got an architecture class, and we’re doing 
shared design projects with students in Norway.’ 
Another faculty member stated ‘I have a small class 
and there’s another fellow in Nebraska with a 
small class, and we are collaborating using the 
Internet. There are all kinds of things like that that 
you can point to that you can say that those are 
things that could not have happened any other way. 
So those are tangible outcomes but you can’t 
measure them.’ 

Another problem identified in trying to create 
measures of the impacts of networking on activities 
like teaching and learning is the lack of good mea- 
sures of these activities, regardless of networking, 
and the lack of existing data on teaching and learn- 
ing in a non-networked environment at some institu- 
tions. ‘We never really measured these outcome 
measures or evaluated the quality of instruction or 
learning or anything anyway. So now we are asking 
how has this proved something that we never mea- 
sured anyway.’ 

Although there are examples of the impacts of 
networking, as described above, measurement of 



these impacts remains very difficult. ‘We are at a 
very immature stage where we really, I think, are 
only getting glimpses of what the future is going to 
hold. And so, it’s going to be very hard. I’d say, to 
measure things, because it’s very foggy just where 
all this is going to go.’ This difficulty in identifying 
and measuring ‘impacts’ from networking is the 
result of an exceedingly complex distributed 
networking environment, a rapidly changing net- 
working infrastructure, and the lack of conceptual 
tools to describe this environment. 

Attitudes, Problems, and Perceptions 

B ased on the various site visits and other data 
collection activities, a number of similar views 
and attitudes toward evaluation and the develop- 
ment of performance measures is evolving. 
Understanding these attitudes and the ‘evaluation 
culture’ at an academic institution is important as 
they will affect the degree to which successful 
ongoing evaluation and use of performance mea- 
sures can be implemented. 

EVALUATION OF NETWORK ACTIVITIES AND 
SERVICES IS A 'GOOD THING' BUT RARELY DONE 

In general, participants agreed with this view. But 
they also agreed that none of them conducts evalua- 
tions regularly. There was an underlying assumption 
that the network is a good thing and that the need 
for it is essential and growing. Therefore evalua- 
tions to determine what’s wrong with the network or 
whether the network is necessary are not needed. 
‘It’s not like I’m going to come out with an earth- 
shattering study that’s going to prove technology is 
worthless and we’re aU going to go back to books. 
It’s not going to happen.’ 

INEQUALITIES OF COMPUTING RESOURCES 

A network administrator described his university as 
a ‘very, very heterogeneous environment. Some col- 
lege units are relatively resource rich some are rela- 
tively resource poor and it’s got more to do with the 
historical situation than with anything that’s evolved 
because of the structural needs of the information 
technology. Addressing that imbalance is going to 
be one of the immediate items on the agenda over 
the next few years, addressing it in some formal, 
systematic way . . . and addressing where the line is 
between central and distributed support.’ He recog- 
nized that having some type of performance mea- 
sures could assist them to deal with this issue, and 
over time, determine the degree to which ‘progress’ 
in equalizing resources had been made. 
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NON-SYSTEMATIC COLLEaiON OF 
NETWORKING DATA 

There was evidence that some data on network per- 
formance are being collected, by different imits 
within the institution and by different people, but 
there is little evidence that the data are being col- 
lected and analyzed in a systematic way or that they 
are being used in planning and decision-making 
related to networking development. Interviewees 
agreed that having a central MIS that identified, 
collected, organized, analyzed, and reported select- 
ed networking statistics would be an important step 
forward and was essential to be able to better plan 
for networking services. They also noted a range of 
problems and issues that would have to be resolved 
if such an MIS were to be established at this partic- 
ular institution. 

REACHING AGREEMENT OF NETWORKING 
TERMS AND EVALUATION PROCEDURES 

Some interesting discussions occurred regarding the 
use of different procedures and different terms for 
the evaluation that had been done at some of these 
sites. There was little confidence that central com- 
puting services could obtain agreement firom other 
stakeholders on campus as to the appropriate defini- 
tions to be used in a larger institutional effort on 
evaluation. One person commented, ‘it would take 
an act of God to reach such agreement.’ Yet, it was 
recognized that imtil campus-wide agreement on 
how best to define key terms occurred, development 
of performance measures would be impossible. 

BARRIERS TO ONGOING NETWORKING 
EVALUATION 

Participants offered a number of reasons far the lack 
of systematic data collection and low priority placed 
on evaluation activities. 

- Some interviewees were suspicious of the goals 
of evaluation . When asked what his response 
would be if his director asked for this type of data, 
one person said, ‘My first question would be, 
“What are you going to do with it?’” The fear of 
how evaluation results might be used prompted 
some to not want to know how well or poorly a 
service was provided. 

- There are ‘power pockets’ throughout the uni- 
versity and a grossly unequal distribution of 
resources. Thus some groups have a vested 
interest in not sharing information about the 
extent of their resources lest they be pressured 
to share those resources. 



- Individuals charged with providing network 
access to a growing, and an increasingly 
demanding, group of users mav not have the 
time and resources to conduct evaluations . 

Their focus is on keeping the network running 
and meeting users’ demands for speed and 
power. As one administrator described it, ‘Life 
is very simple for me as a network planner. I 
need to keep figuring out how to get the best, 
biggest, fastest, cheapest pipe in here possible 
on the presumption that the need for bandwidth 
is going to get bigger and bigger and bigger . . .’. 

- The lack of support (in the form of additional 
resources, a GA to do data coUection. etc.) to 
conduct such evaluations and for some, limited 
knowledge on how to conduct such evaluations 
is also a barrier . Evaluation is ‘just another 
thing to do’ in addition to a range of other 
responsibilities and no additional institutional 
support seems to be provided for such evalua- 
tion. One person indicated that before he would 
feel ‘comfortable’ doing such evaluation, addi- 
tional training would be needed. 

- The lack of incentives to conduct such evalua- 
tions . While generally agreeing that evaluation 
was a good thing, they also noted that there 
were few tangible and direct incentives for con- 
ducting such evaluations. As one person com- 
mented, there was not an institutional mindset 
supporting ongoing evaluations. 

- A lack of faith in the utility and applicability of 
evaluation results . There is an underlying sense 
in many academic institutions that ongoing 
evaluation of services and activities does not 
produce useful results or offer specific recom- 
mendations for how to improve networked 
information services. Further, there is often- 
times no tradition or culture of ongoing evalua- 
tion in the campus setting. 

- The oftentimes confusing distribution of tech- 
nologies and services, and resrxmsibilities for 
managing those technologies and services, com- 
pound the t?roblems associated with evaluation . 
It is not always clear who is responsible for 
what. A number of the participants commented 
that they were unsure who did what in terms of 
networking or were unsure who should be con- 
tacted to solve a particular networking problem. 

Similarly, when a university provides network 
access to information services and resources 
produced by others - such as access to remote 
databases - it is unclear what exactly is being 
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evaluated, services and resources provided by 
the university or by the remote site or by the 
network providers? 

- There are considerable difficulties in defining 
key networking tenns and services in such a 
wav that they can be operationalized for mea- 
surement . Thus, before evaluation can occur, the 
institution may first have to reach agreement on 
how to operationalize key networking activities 
for measurement and clarify policies related to 
networked information services. 

- The recent rapid growth and change in network- 
ing makes evaluation and planning extremely 
difficult . It is not always possible to predict the 
next direction in the technology. I remember 
when someone said, ‘Who needs a laserwriter? 
What are you going to do with that?’ And desk- 
top publishing. Nobody could have predicted 
this stuff. About the time staff get ‘geared up’ to 
evaluate a particular service, it is no longer pro- 
vided or it is out-of-date. 

- Networking infrastructures, services, and admin- 
istrative organization for netwoikingehmee 
rapidly . Evaluation is difficult in such a volatile 
environment. For example, on the day that the 
study team conducted interviews at one site 
visit, the Vice President for Computing 
announced a re-organization of the management 
responsibilities for computing services on cam- 
pus. 

In summary, interviewees agreed they would evalu- 
ate network performance if: someone ‘ordered them 
to do so’; they believed that the results would bring 
them additional resources; they could expect to 
receive additional personnel to conduct evaluations; 
they had training in evaluation methods; and if they 
had the time to conduct evaluations. As one inter- 
viewee said, ‘in an ideal world yes, we would have 
ongoing evaluation; but the reality is we can’t and 
don’t.’ 

Possible Performance Measures 

T he procedures and measures being developed 
for the manual are based on a research effort 
that obtained information, assessments, and input 
from a number of site visits and individuals knowl- 
edgeable about academic computing and network- 
ing. A key finding from this work is the limited 
knowledge and use of performance measures in the 
academic environment. The complexity of the acad- 
emic networked environment imposes some limita- 
tions on the degree to which measures of this envi- 



ronment can be developed. 

Although the manual describes standardized pro- 
cedures for data collection and computing perfor- 
mance measures, the resulting measures are unlikely 
to be comparable across different institutions of 
higher education. The networking infrastructure and 
the marmer in which data are available in different 
institutions vary considerably. Furthermore, differ- 
ent institutions may use different definitions 
for key terms. While these concerns will not hinder 
the use of these measures in one particular institu- 
tion, they will limit the degree to which measures 
can be compared to results at other institutions. 

To some degree, users of the manual may have to 
develop policies and define data collection activities 
within a range of organizational and network con- 
straints. Indeed, some institutions may not currently 
have the capacity to collect the data needed for 
some of these performance measures. In such 
instances, the academic institution will need to first 
determine how best the data can be collected, devel- 
op a system or approach for collecting and analyz- 
ing that data, and develop policies that formalize a 
management information system to insure that the 
data continues to be collected in a regular and stan- 
dardized fashion. 

The research project revealed a number of differ- 
ent views and experiences regarding which types of 
performance measures might be most useful given 
an institution’s particular situation. Thus, the 
approach taken in the manual is to identify and 
describe a core set of measures. Depending on the 
nature of the network, the administrative concerns 
regarding that network, and networking/institutional 
goals and objectives, some of the following mea- 
sures may be more useful for some institutions than 
others. 

The scores that result from these performance 
measures take on greater usefulness when consid- 
ered in the broader context of: 

- Institutional and networking goals and objectives 
at that particular institution 

- Other performance measures of institutional 
activities, services, and participants 

- Various time periods and the amount of change 
on this particular measure over time 

- The amount of resources and the allocation of 
those resources for networking infiraistructure 
and services 

- Factors related to a particular institution, its net- 
working configuration, or other variables unique 
to that institution. 
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In short, value judgements as to whether a score on 
a performance measure is ‘good’ or ‘bad’ are 
dependent on a range of other factors and should 
not be considered in isolation of those factors. 

Finally, it should be noted that the accuracy of the 
actual measures as computed by institutional offi- 
cials will be directly related to the quality of the 
riata they coUect, the use of standardized proce- 
dures, and perhaps, the development of institutional 
policies that define these data collection activities. 

To some degree, these performance measures might 
be best seen as estimates of the extensiveness, effi- 
ciency, effectiveness, or impact of a service or 
activity rather than a precise measure of that partic- 
ular service or activity. Even if these measures are 
best seen as estimates, such estimates are a signifi- 
cant improvement over the very limited set of per- 
formance measures that are currently available and 
being used. 

The performance measures are organized in the 
manual by key areas of assessment. And within 
each area the following measures are currently 
being developed. 

- Users: the number and types of users and the 
frequency with which they use the campus net- 
work 

- Count of Network Users 

by type of user 

- Count of Active Network Users 

by type of user 

- Costs: the total and types of financial resources 
that are expended to operate the academic net- 
work 

- Armual Information Technology Expenditures 

- Information Technology expenditures per 
capita 

- Network traffic: the amount and types of traffic 

' flowing over the academic network 

- Router Traffic as a Measure of Overall 
Campus Network Activity 

- Modem Traffic into the Campus Network 

- Internet Traffic (into and out of the campus) 

- Use: the amount and types of uses made of the 
network 

- Frequency of Network Use 
Percentage of Very Active Network Users 
Percentage of Inactive Network Users 



- Services: the applications and services that are 
made available over the network 

- Online Public Access Catalog Measures 

Number of users using the online library 
catalog 

Number of campus logins to the online 
library catalog 

Number of off-campus logins to the online 
library catalog 

Number of logins to the online library cata- 
log per user 

Cost per user to access the online library 
catalog 

User satisfaction with the networked online 
library catalog 

- Distance Learning 

Number of faculty offering distance learn- 
ing courses 

Number of student enrolled in distance 
learning classes as a percentage of aU 
classes offered 

Distance learning courses as a percentage of 
all courses offered 

Cost per distance learning course 
Technology involved in distance learning 
Student satisfaction with distance learning 

- Support: the types of assistance that network 
officials make available to the users of the net- 
work 

- Help Desk 

Response Time 
Accuracy of Response 
Courtesy of Staff 

Additional measures are also under consideration 
for inclusion in the manual. Space does not permit a 
detailed description of these proposed measures, 
how they have been operationalized, and procedures 
for data collection and analysis. The draft perfor- 
mance measures manual contains such information. 

In developing these measures the study team 
found that oftentimes the academic institution 
would first have to deal with and resolve a range of 
issues before the performance measures could be 
computed. For example, the measures ‘count of 
network users’ (CNU), ie. the number of identified 
email accounts with access to the campus network, 
and ‘count of active network users’ (CANU), ie. the 
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number of email accounts that have logged onto the 
network during a one-month period, cannot be com- 
puted until the following issues are resolved: 

- Defining ‘the network’ . For purposes of these 
counts we recommend that the campus network 
be defined as those telecommunications services 
and resources over which the academic institu- 
tion has primary responsibility and control. 

- Defining users. There may be more accurate 
means to identify network users than email 
accounts, depending on the record-keeping tech- 
niques used at a particular institution, eg. user 
IDs, official registrations, or payroll records. 

- Including distributed computing accounts . For 
many institutions there are multiple servers with 
their own administration and email accounts. 
Thus, a decision must be made whether to 
include only centrally administered email 
accounts or to include email accounts from dis- 
tributed servers in the CNU and CANU. 

- Purging inactive accounts . The accuracy of the 
CNU wUl depend on the institution’s policy 
regarding purging inactive accounts. Policies 
should be in place that regularly purge accounts 
from the files for those who are no longer legal 
institutional members. 

- Defining users in the campus networking com- 
munity . There may be significant numbers of 
individuals with email accounts on the campus 
system who are ‘guests’ and do not belong to 
the campus community but use the network, eg. 
students who graduate but continue using the 
network for mail and other applications. 
Decisions must be made to consistently count 
the ‘bona fide’ members of the campus net- 
working community. 

Further, it may be unclear how to determine 
who is ‘faculty’ or ‘students’ or ‘staff’ or other 
‘types’ of users. Definitions for such user types 
may need to be agreed upon if CNU and CANU 
are to be broken down by type of user. 

- Defining what coastitutes an ‘active user’ . For 
purposes of CANU we have recommended that 
an active user is one who has shown any net- 
work activity on his/her email account in a pre- 
ceding one-month period. Some institutions 
may wish to use a different time period to 
define ‘active user’. 

- Multiple email accounts . Some individuals on 
campuses with multiple servers may have multi- 
ple email accounts. Thus, the number of email 



accounts is not the same as the number of indi- 
viduals with email accounts. The level of analy- 
sis for ‘email accounts’ is different from ‘indi- 
viduals with email accounts.’ For the CNU and 
CANU some institutions might wish to sample 
users to determine the average percentage who 
have multiple accounts to estimate the number 
of individuals with email accounts as opposed 
to the number of email accounts. 

Until such issues are resolved campus-wide, it 
would be impossible to produce an accurate count 
of network users and active network users. 

For each of the performance measures, the manu- 
al provides an operationalized definition, issues that 
may have to be resolved prior to obtaining data to 
produce the measure, data collection procedures, 
and suggestions for developing related measures. 
The actual measures to be included in the final ver- 
sion of the manual may change from those identi- 
fied above depending on field testing of the manual 
during the FaU, 1995. 

Next Steps 

T his project is scheduled for completion in 
December, 1995. At that time the authors wiU 
submit a final report to the funding agency summa- 
rizing project activities. The study team also will 
produce a performance measures manual for public 
distribution. The manual will have the following 
sections: 

- Introduction to using the manual 

- Quantitative performance measures 

- Example network user survey 

- Guidelines for collecting, organizing, and report- 
ing anecdotal and other qualitative assessments. 

The manual is a beginning effort to provide stan- 
dardized guidelines to assist academic institutions to 
assess their academic networked environment. It has 
the following objectives: 

- Describe a core set of performance measures that 
assess the academic networked environment 

- Provide procedures for collecting and analyzing 
the data needed to produce these performance 
measures. 

- Identify and discuss issues and problems related 
to data collection needed for computing these 
performance measures. 

- Encourage academic institutions to engage in a 
regular program of ongoing evaluation and 
assessment of their computing networks. 
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The manual can assist network managers and higher 
education decision-makers to improve the useful- 
ness and quality of their networks and better meet 
the needs of network users. The manual is currently 
in draft and is constantly being revised and expand- 
ed in response to ongoing data collection activities. 

During the Fall of 1995 the study team will con- 
tinue an iterative process of revising the manual, 
field-testing it at appropriate academic sites and 
with knowledgeable individuals, and then revising/ 
editing the manual. A final assessment and review 
of the manual will take place at the Fall, 1995 meet- 
ing of the Coalition for Networked Information, to 
be held in Portland, Oregon, USA, October, 1995. 
Based on this assessment the study team will pro- 
duce a final version of the manual for release in 
early 1996. 

Importance of the Project 

T he numerous initiatives associated with devel- 
oping the National Information Infrastructure in 
the USA (eg. National Information Infrastructure 
Advisory Council, 1995) and throughout the world 
with the evolving Global Information Infrastructure, 
have thrust electronic networked computing into a 
new arena and into a new teaching, learning, and 
research environment. The uses and applications of 
networking and the Internet continue to grow rapid- 
ly, roles and responsibilities of key stakeholders in 
the networked enviromnent become increasingly 
blurred, a range of policy issues (eg. acceptable use, 
intellectual property rights, and equitable access) 
and questions regarding the effectiveness, efficien- 
cy, and impact of the network in academic institu- 
tions continue to be poorly defined and addressed 
(Heterick, 1994). 

To a large extent, proponents for enhancing the 
academic networked environment have said, ‘trust 
me, trast me ... , access to and use of electronic 
networks improves the quality of education here at 
our institution.’ But the reality is that evidence to 
support such assertions is either non-existent or 
anecdotal. In times of budget cuts and institutional 
retrenchment (such as we are seeing today), faculty, 
librarians, administrators, and academic computing 
service providers find it increasingly difficult to jus- 
tify expenses for purchasing network technology, 
supporting network services, developing training, or 
demonstrating that such networks really have some 
impact on the educational imperatives of the institu- 
tion. 

Based on a number of research projects on the 
development of networked services and digital 



libraries in academic settings, Covi and Kling 
(1995, p.5) conclude: 

‘Our early observations suggest that universities 
appear to be steadily drifting into more intensive 
digital investments with little managerial over- 
sight about the extent to which their investments 
are effective or efficient, adequate or fiugal.’ 

Given the size and extent of such investments, and 
the widespread financial difficulties many institu- 
tions of higher education are experiencing, such a 
conclusion is most troubling. Until we have a better 
conceptual framework describing the ‘academic net- 
worked environment’ and performance measures to 
assess interactions and services within this environ- 
ment, we will only guess at what seems to work 
well and why. We will only be able to guess at 
which strategies have had the greatest impact, for 
example, on learning. And, we will only be able to 
guess at how best to design better networked ser- 
vices in the future. 
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Appendix A 

5/15/95 

Announcing A New CNI Electronic 
Discussion Group: 
IMPACTS@CNI.ORG 

IMPACTS is a public, moderated computer forum 
that will discuss the effects of networked informa- 
tion technologies and services on academic institu- 
tions and the measurement of those effects. The 
IMPACTS forum provides the opportunity for 
researchers, academic administrators, academic 
computing personnel, network specialists, faculty, 
students, academic staff, and others to exchange 
views and information related to this topic. 
IMPACTS will inform the research project 
Developing Performance Measures to Assess the 
Impacts of Internet Networking on the Academic 
Institution, funded by the U.S. Department of 
Education. Objectives for the discussion forum are 
to: 

- Provide a forum to discuss applications and 
issues in the use of performance measures for 
assessing academic networking 

- Obtain input from interested individuals about 
the project and various project documents and 
position papers 

- Share information on related projects or other 
institutional efforts to develop and use perfor- 
mance measures to assess academic networking. 

IMPACTS May Include Topics Such 
As 

- What information technologies and services 
comprise networked information, and to what 
degree are these similar across various academic 
institutions? 

- Who are the “users” of networked information 
within the academic setting and how might we 
develop a typology of such users? 

- What are the organizational structures used in 
academic institutions to provide networked 
information services? 

- What are the key factors that appear to affect the 
overall success of the networked envirorunent in 
an academic setting? 

- What measures can be developed to assess the 
impacts of the networked information services 
and resources on the academic environment? 
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To SUBSCRIBE to the IMPACTS Forum 

To subscribe to IMPACTS, send the following email 
message to LISTPROC@CNI.ORG: subscribe 
IMPACTS <your first name> <your last name> 

To SEND MAIL to the IMPACTS 
Forum or Obtain Additional 
Information 

To participate in the list discussion, please send 
your mail to: IMPACTS@CNI.ORG. The 
IMPACTS list moderator is Kristen Eschenfelder 
<kreschen@mailbox.syr.edu> who should be con- 
tacted for matters regarding the list. For additional 
information about the project contact the Co-princi- 
pal Investigators: Dr. Cynthia L. Lopata 
(cUopata@mailbox.syr.edu) or Dr. Charles R. 
McClure (cmcclure@mailbox.syr.edu). School of 
Information Studies, Syracuse University, Syracuse, 
NY 13244(315-443-2911). 

Discussion 

John Sumsion, Library & Information Statistics 
Unit (LISU): Two things you have not mentioned: 
one is the possibility of coimting the amoimt of 
information provided through the Internet by the 
information providers who charge for it. The other 
is a kind of citation analysis. Is there any future in 
asking people in their research papers to indicate 
whether they have acquired the information through 
the Internet as opposed to printed sources. 

Charles McClure: To answer you second point first: 
already I see that happening. I am the founder of the 
journal Internet research in which we regularly cite 
electronic sources. There are a number of manuals 
that give the correct style on citation of electronic 
information. Citation coimts would be very good 
idea for finding out where information is coming 
from. 

Your first question on tracking the amount of 
information coming from providers that sell it is 
really tough. We keep coming back to privacy 
issues. You do not know whether the bits coming 
through are fee-based or free-based. Unless we 
could get providers somehow to mark it, which in 
my experience is not very pleasant, I am not sure 
what to do about that. 

Karin De Jager, University of Cape Town: I am 
interested in your giving readings for students 
online. Have you considered copyright issues? 



Charles McClure: Yes I certainly have. A lot of the 
material I use is by people I already know and can 
call up on the phone for permission to use, without 
going through their publisher. Ultimately, I have 
found it to be no more difficult to put a reading up 
electronically and scan it than it is to put it in a 
reader. Those of you in library land should begin 
worrying about people like me in the faculty who 
will say: handle this problem for me. Frankly, my 
library doesn’t handle it so my graduate student 
does it. 

In terms of pushing copyright limits, in terms of 
educational uses and so on, if you put yourself on 
the other side, it is really not pleasant to see a full 
chapter of one of your books posted electronically 
out on a Web site, for example, which has a copy- 
right on it. I am on a discussion net called Compriv 
- Commercialisation and Privatisation of the 
Internet - and was recently reading a discussion 
when I realised I was reading my own material. 
Someone had downloaded a report, changed a cou- 
ple of things, put their name on it and sent it up. We 
in academic land understand this notion of intellec- 
tual property. Go down to the town and they don’t 
have a clue what that means. Those issues of copy- 
right and intellectual property rights are bmtal. 

They are not going to go away. They are going to 
take some serious work. I don’t have the answers. 

Michael Carmel, S.W. Thames Regional Library 
Service: It seems your talk has been in two parts. 
You started with money problems and the difficulty 
of justification, you then went on to performance 
measures, In between you said the networked envi- 
ronment is going to happen. 

Charles McClure: What is going to happen is more 
and more people using the Net without us being 
able to say how they are using it or why. 

Michael Carmel: So if it is going to happen, why do 
we need to measure it in order to justify it. 

Charles McClure: Great question. Here’s why. 
Think about what’s going to happen when 2000 
fireshmen get on your campus tomorrow and they all 
want to get connected to a Web server. Do you have 
the bandwidth to do that? 

Michael Carmel: I am not on an academic campus. 

Charles McClure: But you imderstand what I am 
saying. If more and more people are getting on, we 
don’t know what kinds of netwoik services to pro- 
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vide. We don’t know how to plan for them. For pur- 
poses of planning services to meet user needs, we 
have to know what is going on. What’s being used 
and how well they use it. Our campus- wide infor- 
mation system, for example: I turn on my computer 
and get this menu - and I am constantly reassured 
by computing that this is a user-friendly manual. It’s 
not. Until they understand what constitutes user 
friendly, through evaluation and ongoing assess- 
ment, we wUl continue to get non-user-friendly 
user-friendly interfaces. 

Don Revill, LiverpoolJohn Moores University: 
What do you think about demand reduction strate- 
gies? Our computing people see the network as its 
own justification. They see us as trying to control it, 
codify it, classify it, just like librarians. Also, my 
senior managers don’t understand computers. My 
Vice Chancellor gets his secretary to see to his 
email. They see it as heap powerful ju-ju. Have you 
any comments on that? 

Charles McClure: I understand demand reduction 
strategies. You really don’t want ten million people 
waiting in queue at the help-desk. What I proposed 
at one of the universities, was for them to do more 
training, which would reduce the number of ques- 
tions. When you do that, you increase people’s 
knowledge to where they’re dangerous. They want 
to know more. The notion of demand reduction 
strategy is an interesting one. It’s problematic. 

Second comment: it is a very real problem that 
the people who demand justification and perfor- 
mance measures are exactly the ones who don’t 
know anything about the Net. They never use it. 
Their secretaries use it. Increasingly we have to say 
to people: excuse me, you don’t know how to read 
email, watch me. I wUl train you. I will help you. 
That is something the library community can do. 
Training and education are critical and are some- 
thing the library community could do well. We 
could really carve out a niche for ourselves in this 
networked environment. But again, that is just one 
more little leaf on top of the salad. We are anaUy 
retentive, compulsive, type A people. One of the 
things performance measurement is supposed to do, 
is to help you prioritise. I don’t think it has done 
that. We have widened our playing cards even more. 
We have got to set priorities. 
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